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Validating Career-Readiness Features in High 
School Assessments? 


Jenny C. Kao, Nichole M. Rivera, Brettany Clemens, and Li Cai 


CRESST/University of California, Los Angeles 


Abstract: This report is the fourth in a series considering career-readiness 
factors within existing high school assessments. The primary goal of this 
study was to provide a preliminary validation of the career-readiness 
features identified in prior reports by exploring how different participant 
groups with different levels of experience in the career/vocational world 
perform on a selection of test items with high numbers of career- 
readiness features. Two exemplar careers, emergency medical technician 
(EMT) and web developer, were targeted for participation in this study. A 
total of 103 high school students, 111 community college students 
studying to become either EMTs or web developers, and 84 working 
EMTs or web developers participated in the study. A selection of ELA and 
math test items rated for career-readiness features were adapted into an 
18-item test booklet. As expected, results indicated that workforce 
individuals scored significantly higher on the test than community college 
students, who in turn scored significantly higher on the test than high 
school students. These results suggest that having added experience in 
their field may lead to refining certain career-readiness skills found in 
high-school-level content-based assessments. Preparation for such 
assessments can help high school students prepare for college and/or a 
career, and inferences for both college and career readiness can be 
drawn from test performance. 


Introduction 


The national high school graduation rate is at an all-time high, with the national 
graduation rate exceeding 84% for the class of 2016 (EDFacts, 2018). However, if these 
graduates decide not to continue their education beyond high school, they can expect to earn 
about half of what their college-educated peers will earn annually. Many graduates lack the 
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knowledge, skills, and attributes necessary to compete and succeed in fields such as advanced 
manufacturing; energy; healthcare; information technology; and science, technology, 
engineering, and mathematics, or STEM (Council of Chief State School Officers, 2013, 2014; 
O*NET Online, n.d.). There is a shortage of qualified job candidates in many high-demand, high- 
skill, and high-wage career categories (Executive Office of the President of the United States of 
America, 2014). Prior to 2012, American workers most often held a high school diploma but had 
no college experience. Since 2012, people with some college or an associate’s degree have 
made up the largest share of the U.S. civilian labor force (Brundage, 2017). By 2020, it is 
expected that 65% of all jobs will require postsecondary education and training (Carnevale, 
Smith, & Strohl, 2013). In response, the Department of Education has emphasized the 
importance of college and career readiness. 


One contributor to the shortage of qualified job candidates may be an underlying deficit 
in college and career readiness in the K-12 and postsecondary educational system. Classrooms 
still remain largely unconnected to the needs of employers and success in postsecondary 
education. With a large percentage of students requiring remediation in basic math and English 
skills prior to their freshman year, there is evidence that students are not equipped with the 
skills needed to begin college-level work (California State University, 2017). As noted in a recent 
report on the state of work readiness, higher levels of education do not always guarantee work 
readiness (Mattern et al., 2014, p. 4). 


College and Career Readiness 


The terms “college ready” and “career ready” are often used interchangeably, without 
grounding in evidence or validity. Furthermore, there is no federal definition of “college and 
career readiness.” Rather, the government defines standards as “goals for what students 
should know and be able to do while learning academic content” (U.S. Department of 
Education, n.d.). Determining what college ready and career ready mean is left for the states. 
Content areas and academic achievement are typically used to define college and career 
readiness, but research suggests other personality variables such as conscientiousness and 
other noncognitive features may have predictive power (Mattern et al., 2014). There is growing 
interest in social and emotional learning (SEL) as it relates to career readiness (CASEL, 2017). 
These are processes through which students acquire and effectively apply the knowledge, 
attitudes, and skills necessary to understand and manage emotions and set and achieve goals 
(Dymnicki, Sambolt, & Kidron, 2013). 


According to Mishkind, as of 2014, 33 out of 36 states and the District of Columbia used a 
single definition to span college and career readiness. In a recent policy summary, Mishkind 
categorized the skills reported by these states into six common categories: (a) academic 
knowledge; (b) critical thinking and/or problem solving; (c) social and emotional learning, 
collaborations and/or communication; (d) grit/resilience/perseverance; (e) citizenship and/or 
community involvement; and (f) other additional activities (Mishkind, 2014, pp. 1-3). Similar to 


Mishkind’s findings, the set of career-readiness features in this study could fall into such 
categories and span more than one. For example, our mathematical reasoning feature could be 
categorized as academic knowledge but could also be categorized as a critical thinking/problem 
solving skill. Our organizing, planning, and prioritizing work feature could be aptly grouped 
under SEL. 


Unlike some other states, California does not have a clear definition of college and career 
readiness. The California Department of Education (CDE) previously outlined college- and 
career-ready standards. In 2014, the CDE published these 12 standards, known as Standards for 
Career Ready Practice. Some examples include “Apply technology to enhance productivity,” 
“Practice personal health and understand financial literacy,” and “Act as a responsible citizen in 
the workplace and the community.”* While admirably inclusive, these standards may have 
challenges for implementation and measurement. In 2015, the CDE launched the California 
Career Readiness Initiative with “21 key objectives to support, sustain, and strengthen Career 
and Technical Education (CTE) in the State” (CDE, 2018b). Additionally, in February 2018, the 
CDE released guidelines for SEL which are “fundamental to academic success” and necessary to 
“truly” ensure college and career readiness. These guidelines include the ability to (a) set and 
achieve positive goals; (b) feel and show empathy for others; (c) establish and maintain positive 
relationships; (d) make responsible decisions; and (e) understand and manage emotions (CDE, 
2018a). 


As a part of California's new accountability system, the CDE has specified five separate 
paths (A-E) which may indicate whether a student is college or career ready (CDE, 2018b; p. 
55). California has been part of a consortium implementing Smarter Balanced for standardized 
testing, ensuring students are ready for college success. Three of these pathways include 
achievement on Smarter Balanced Assessments. (See Appendix A for additional information on 
the five ways students can show college and career readiness as defined by the CDE.) In 
transitioning to Common Core State Standards, Smarter Balanced developed their assessments 
in consideration of college and career readiness (Smarter Balanced Assessment Consortium, 
2015, p. 8). 


In 2011, the U.S. Department of Education noted that under the reauthorization of the 
Elementary and Secondary Education Act (ESEA), states have developed assessments aligned 
with their standards, but they often did not adequately measure student knowledge and skills 
and provide timely feedback for parents and teachers. States were directed to develop and 
administer assessments which accurately measure student learning on college- and career- 
ready standards (U.S. Department of Education, 2011). A number of colleges are currently using 
scores from Smarter Balanced as evidence that students are ready for credit-bearing entry-level 
college courses without the need for remediation (Smarter Balanced Assessment Consortium, 
2015). If scores accurately reflect criteria for career and college success, this may offer long- 


?Prepared by the California Department of Education. The complete practices are available at 
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term value to students, schools, and ultimately, the workforce as a whole. It is with this goal 
that we undertook career-readiness feature analysis of existing test items. Our companion 
reports on career-readiness features discuss this process in more detail (Choi, Kao, Rivera, & 
Cai, 2018; Kao, Choi, Rivera, Madni, & Cai, 2018; Madni, Kao, Rivera, Baker, & Cai, 2018). 


Career Growth and Education 


In terms of job growth, there has been a shift toward skilled occupations which typically 
require some postsecondary education. Specifically, the greatest growth since the Great 
Recession of 2007 has been in management and healthcare occupations (Carnevale, 
Jayasundera, & Gulish, 2016). Carnevale et al. posited that economic recovery since the 
recession in the United States “has divided the country along a fault line demarcated by college 
education” (2016, p. 1). Those with at least some college education continue to gain shares in 
the job market, while blue-collar jobs in sectors like construction and manufacturing continue 
to lose millions of jobs. Jobs requiring an associate’s degree will grow faster than those 
requiring a bachelor’s degree through 2020 (U.S. Bureau of Labor Statistics, as cited by 
California Community Colleges, n.d.). Thus the current research aimed at exploring two careers 
as exemplars, emergency medical technicians (EMTs) and web developers. These two jobs 
project high growth and typically require some postsecondary education at the entry level, but 
not necessarily a bachelor’s degree (U.S. Bureau of Labor Statistics, n.d.-a, n.d.-b). In the 
following subsections, we describe these two careers in brief. 


EMTs. The U.S. Bureau of Labor Statistics forecasts strong growth for EMTs and 
paramedics in its Occupational Outlook Handbook. It reports a projected 15% change in 
employment from 2016 to 2026, higher than the average of 7% for all occupations (U.S. Bureau 
of Labor Statistics, 2017b). As of 2017, the nationwide median annual salary for this occupation 
was $33,380 (U.S. Bureau of Labor Statistics, 2017a). 


Emergency medical services (EMS) are a vital part of the healthcare system, providing 
transportation and prehospital care in emergency and trauma situations. There are both public- 
sector and private-sector EMS providers. The private-sector EMS is composed of mostly for- 
profit companies, with two large companies operating 48% of private ambulances in California. 
Public sector EMS includes fire departments and municipal and county EMS services (Jacobs, 
Heller, Waheed, & Appel, 2017). 


In California, there are three levels of EMS practitioners: EMTs, advanced EMTs, and 
paramedics (California Emergency Medical Services Authority, 2017). For this study we focused 
on EMTs. EMTs provide basic, noninvasive interventions in prehospital settings and transport 
patients to the hospital (National Association of Emergency Medical Technicians, 2018). 


EMT training. Becoming an EMT or paramedic requires training, certification, and 
licensing. Training is offered at community colleges, universities, hospitals, and technical 
schools, as well as EMS, fire, and police academies. The National Registry of Emergency Medical 
Technicians (NREMT, 2018) certifies EMTs at the national level, but EMTs are licensed by states. 


To qualify for NREMT certification, a student must complete a state-certified education 
program and pass the national exam (U.S. Bureau of Labor Statistics, 2017b). 


Some programs are short-term and intensive, while others are more flexible and longer 
term (National Association of Emergency Medical Technicians, 2018). In California, education 
programs are certified by the California Emergency Medical Services Authority (EMSA). EMTs 
require 146 hours of didactic and skills training and 24 hours of clinical training, for a minimum 
of 170 hours of training. Paramedics require 450 hours of didactic and skills training, 160 hours 
of clinical training, and 480 hours of field internship, for a minimum of 1,090 hours of training 
(California Emergency Medical Services Authority, 2018a). After completing an approved EMT 
training program and passing the NREMT exam, one can apply for certification at a local EMS 
agency (California Emergency Medical Services Authority, 2018b). California EMT certifications 
are valid for two years and require ongoing continued education to renew the license. 


Web developers. At the time of this report, the BLS projected a 13% change in 
employment from 2016 to 2026, which is much higher than the average of 7% for all 
occupations (U.S. Bureau of Labor Statistics, 2017c). In May 2016, the nationwide median 
annual salary for this occupation was $66,130 (U.S. Bureau of Labor Statistics, 2017a). 


Web developers create and maintain websites and their infrastructure. Some web 
developers program aspects of the website that affect how it appears on the user’s screen. 
These are referred to as front-end web developers. Others are responsible for the logic of the 
website, which is accessed indirectly by users interacting with the front-end of the website. 
These are called back-end developers. Some web developers program both the front-end and 
the back-end aspects of websites and are hence called full-stack web developers (U.S. Bureau of 
Labor Statistics, 2017c; Wales, 2014). While web developers may work at companies, the field is 
composed of a large number of freelancers (Hadad, 2018). 


Web developer training. At the time the study was undertaken, the BLS listed an 
associate’s degree as the typical education for entry-level web developers. Specific training or 
skill requirements for web developers vary based on the needs of an employer. A bachelor’s 
degree in computer science, programming, or a related field may be required for more 
specialized web developer positions, as well as to advance into project management positions 
(U.S. Bureau of Labor Statistics, 2017c). 


Another growing pathway beyond traditional degree programs is skill-based “bootcamps.” 
Web developers require constant skill development. Bootcamps generally allow rapid entry into 
the field or allow workers to enhance their skills (Wilson, 2017). On the plus side, these 
programs offer skills that are focused on careers and current in-demand skills, require less time, 
and bridge high-tech workers with employers. Negatives include less emphasis on theoretical 
knowledge, no degree awarded, geographical availability, upfront payments, and high levels of 
attrition (Wilson, 2017). 


One interesting question may surround the types of applicants these programs attract. A 
recent study noted that despite the well-known gender disparity in the field of computer 
science, 43% of postbaccalaureate “coding bootcamp” participants were women in 2016, 
significantly more than the 26% of computer science professionals overall (Siebel, 2018). 
Further, these programs may attract older participants beginning a second career. Little is 
known about the issues and barriers these students face (Thayer & Ko, 2017). 


Purpose of the Study 


The present study was one strand in a larger project exploring career-readiness indicators 
within the context of K-12 assessments. As described in our earlier report (Kao et al., 2018) we 
developed an initial set of career-readiness skills/features which were then used to rate a 
selection of Smarter Balanced test items. The purpose of the present study was to provide a 
preliminary validation of career-readiness features in high school assessment items by 
exploring the performance of different participant groups with different levels of experience in 
our two exemplar careers (EMT and web developer). A refined set of test items with high 
numbers of career-readiness features was administered to high school students, community 
college students enrolled in a certificate or degree program in one of the two exemplar careers, 
and adults working in one of the two exemplar careers. We were also interested in whether 
other background factors, including noncognitive factors, played a role in their performance. 
Because of the quasi-experimental nature of the design, results of this study are meant to be 
exploratory and inform future work. The following exploratory research questions guided this 
study: 

1. Do participants with more experience in a career/vocation perform higher on test 

items that contain high numbers of career-readiness features? 


2. What background factors play a role in the performance on test items that contain 
high numbers of career-readiness features? 


We expected that the participants with work experience would perform higher than the 
community college students, who in turn would perform higher than the high school students. 


Method 


Participants 


A total of 298 participants were included in the analyses for this study: 103 Grade 12 high 
school (HS) participants, 111 community college (CC) students, and 84 working adults 
(hereinafter: workforce participants or WF). Of the 111 community college students, 64 were 
enrolled in EMT programs and 47 were enrolled in web development programs. Of the 84 
workforce participants, 45 were EMTs and 39 were web developers. Table 1 provides general 
demographic information of the participants by subgroup and by career domain (i.e., EMT and 
web development). 


Table 1 


Demographic Information by Subgroup and by Career Domain 


Community college Workforce 
High Web Web 
school EMT developer EMT developer Total 

N 103 64 47 45 39 298 
Age range 16-18 18-45 19-56 19-42 20-50 16-56 
Age mean (SD) 17.0(0.4) 22.6(4.8) 27.8 (7.6) 24.6(5.0)  33.8(9.8) 23.2 (7.9) 
Female 52.9% 23.8% 25.5% 13.6% 25.6% 32.9% 
Male 47.1% 76.2% 72.3% 86.4% 74.4% 66.8% 
Nonbinary 0 0 2.1% 0 0 0.3% 
American Indian or 0 0 0 0 0 0 
Alaskan Native 

Asian or Pacific Islander 3.9% 7.9% 13.0% 14.0% 53.8% 14.3% 
Black or African American 16.5% 3.2% 2.2% 2.3% 5.1% 7.8% 
Hispanic or Latino/a 47.6% 22.2% 30.4% 23.3% 10.3% 31.0% 
White or Caucasian 10.7% 42.9% 19.6% 51.2% 12.8% 25.2% 
Other 1.0% 7.9% 2.2% 0 5.1% 3.1% 
More than one 20.4% 15.9% 32.6% 9.3% 12.8% 18.7% 


race/ethnicity 


Note. There were missing cases in some categories; percentages were computed on valid cases. Participant write- 
in responses for race/ethnicity included: Arabic, Armenian, East Indian, Guyanese, Indian, Italian, Jordanian, 
Middle Eastern, Native American, North African, Pakistani, Persian, Sri Lankan. 


Participant recruitment varied depending on the subgroup and the career domain (i.e., 
EMT or web developer). Information on participant recruitment is detailed in the following 
subsections. 


High School Recruitment 


A total of 103 Grade 12 students representing four schools (Schools A, B, D,? and E) in 
Southern California participated in the study. Effort was made to recruit participants from 
schools with varied demographic and socioeconomic background profiles that roughly matched 
demographics across the state. School A is a small charter school in a large city, serving Grades 
6 through 12 (n = 17 participants). Schools B and D are large public high schools in small cities, 
serving Grades 9 through 12 (n = 25 and n = 31, respectively). School E is a large public high 


3School C participated in the cognitive lab interviews discussed in a prior report (Madni et al., 2018), which also 
included Schools A and B. School C did not participate in the present study. 


school in a large city, serving Grades 9 through 12 (n = 30). Students from School B were 
enrolled in the Advancement Via Individual Determination (AVID) program. Students from 
Schools D and E were enrolled in Career Technical Education. With appropriate permissions 
from the school districts, principals, and teachers, where applicable, students who volunteered 
and returned signed parent permission forms participated in the study. 


Community College Recruitment 


As general criteria for recruitment, community college students had ideally completed at 
least 20% of their program. Participants who had recently graduated but had not started 
working in their field were also included. This requirement helped ensure that participant 
experience in the domain was meaningfully different from high school and workforce groups. 


In order to ensure a diverse sample, a list of potential community colleges was initially 
created using information from the California Registry of Community Colleges. Community 
college recruitment for EMT and web development students happened simultaneously. 
Because of school-specific programs and degrees, some sites contributed participants to more 
than one participant subgroup and some sites contributed to only one. Students from a total of 
four external sites (Sites F, G, H, and J, described further below) participated. For the purposes 
of this study, all participants in this group regardless of program or site (certificate or degree) 
are referred to as “community college.” 


EMT. EMT students were recruited from either Site F or Site G. Site F is a small community 
college located in a small city in Southern California, which also has a program for web 
development. The EMT program consists of two courses taken sequentially over two semesters. 
The first course is an introduction to the EMS system and the second course contains the 
majority of the content. 


Site G is a non-community-college-based training program with more than one campus in 
a large city, and offers various emergency medicine certificates in a variety of course formats 
(including online). The EMT program consists of four courses taken as corequisites which could 
be taken in either a month-long accelerated format, or a longer, hybrid online and classroom 
format. Sixty-five EMT students participated in the study, however one participant was 
excluded from the final analyses because of prior work experience as an EMT. Thus, the final 
number of community college EMTs included in the analyses was 64 (21 from Site F and 43 
from Site G). 


Web developer. Web developer students came from one of three sources: Site F is the 
same small community college in which we also recruited EMT students described above; Site H 
is a large community college located in a large suburb; and Site J is a “coding bootcamp,” an 
intensive, short-term skills training program for web development. 


The field of web development is broader and less defined than the field of emergency 
medicine. Because of the breadth and diversity within the field of web development, 


researchers determined that it would be impractical to require programs to have every skill and 
ability named under O*Net’s description of web developers. Thus, in order to select programs 
to target for the present study for web development, researchers met with subject-matter 
experts and developed criteria based on which features were most critical to the job 
description and likely to appear in a syllabus or course description. For example, while working 
on a team or managing people may be an important component of a web development job, it is 
not job specific or likely to be taught as a component of a limited degree or certificate. 


Based on this foundational research, the educational programs in web development were 
targeted based on the following criteria: Programs were named “web development” or were in 
an appropriate department (e.g., computer science) and included coursework in 
(a) programming language(s); (b) web design/browser interface; and (c) back-end functions. 
Additionally programs were required to include some in-person coursework (not exclusively 
online) and be a part of a certificate or degree program not intended to exceed two years in 
length. Two researchers evaluated each program curriculum based on the above criteria. 
Programs which were labeled web development or met two of the three criteria were targeted 
for recruitment. Additionally, within the field of web development and technology, another 
common educational route emerged in the form of coding bootcamps. Given their growing 
popularity in this sector, researchers targeted local bootcamps, all of which met the criteria 
described above. One of these bootcamps was included in the study. 


At Site F, the program is a general computer-related Associate of Science degree with an 
emphasis in web design. This degree includes courses covering web page programming and 
design, CSS, Java, C++, Dreamweaver, Flash, Adobe Photoshop, and Adobe Illustrator. Site H’s 
web development program culminated in a certificate. The required courses for this certificate 
covered topics such as HTML, CSS, Javascript, web servers, and web programming. Site J, the 
bootcamp, offered a full-time curriculum lasting 14 weeks. At the time of our study, students 
learned front-end development and back-end development with Python, and chose two of the 
following: MEAN, Ruby on Rails, Java, C#/.NET, or iOS. 


Fifty-eight community college students participated as web developer students in this 
study. Despite caution in recruitment, 11 participants did not meet certain requirements based 
on examination of background surveys and were excluded from the final analyses. Six were 
excluded because they reported studying programs unrelated to web development (such as 
building administration and cybersecurity). Two were excluded because they had prior work 
experience as EMTs. Two others were excluded because they currently held part-time jobs in 
web development. Another person was excluded because not enough information was 
provided to suggest that he was enrolled in the correct program. Thus, the final number of 
community college web development students included in the analyses was 47 (10 from Site F, 
18 from Site H, and 19 from Site J). 


Workforce Recruitment 


For both career domains, we initially required that participants were (a) currently working 
as either an EMT or web developer, and (b) had at least six months of work experience in the 
respective domain. However, due to the various challenges of recruiting working adults, minor 
exceptions were made to the first criterion, in that those unemployed for under a month were 
included. 


EMT. Workforce EMTs were recruited from a variety of sources. An additional criterion for 
EMTs was possession of a current California EMT license. Almost half (n = 24) came from a full- 
time paramedic program at Site H described above. The paramedic program was in its second 
day, and through informal conversations with the instructor and the participants, all students 
had recently quit their jobs as EMTs in order to enroll. A current EMT license and over 1,200 
hours of experience working as an EMT was required for admission to the program. Because of 
the program enrollment criteria, and because the program had only just begun, we felt it was 
appropriate to include these participants in our study. 


Other participants were recruited through program alumni listservs and EMT 
recertification courses, with the assistance of program personnel. Four participants were 
affiliated with Site F as instructors. Additionally, ambulance services, hospitals, and other 
businesses employing EMTs were contacted. 


Fifty-one people participated in our study as workforce EMTs. Upon further examination 
of background surveys, six participants were excluded from the final analyses because of job 
titles which would not provide the necessary experience in the domain. These job titles 
included emergency department technician, emergency room coordinator, diver medic, ocean 
lifeguard, and U.S. Forest Service. Thus, the final number of participants included in the final 
analyses for all workforce EMTs was 45. 


Web developer. Working web developers were recruited through program alumni 
listservs, general technical listservs, technology-related events, and affiliated businesses. We 
also hosted an event advertised online through tech-related social media. A few were also 
referred to us by a technical employment staffing agency. Criteria for recruitment were similar 
to that of student web developers, in that participants needed to have experience in at least 
two of the following: programming language(s), web design/browser interface, and either 
front-end or back-end functions. 


Forty-five people participated in our study as workforce web developers. However, upon 
further examination of background surveys, six participants were excluded from the final 
analyses. Three were excluded because their job titles appeared unrelated to web development 
(i.e., software engineer, computer support, computer science teaching assistant). Two were 
excluded because they had less than six months of work experience. One was excluded because 
he had been unemployed for more than six months (based on informal conversation with 
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participant). Thus, the final number of participants included in the final analyses for workforce 
web developers was 39. 


Instruments and Measures 


Test instrument. An 18-item test instrument was adapted from Smarter Balanced 
summative computer adaptive test items into a paper-and-pencil-based administration. All 
items were from either 2015 and/or 2016 Grade 11 test administrations. Half of the items 
covered English language arts (ELA) while the other half covered mathematics. Two forms 
(Form E and Form M) were created to counterbalance the content areas and control for the 
effect of content order, as well as discourage cheating. Form E began with English language arts 
items while Form M began with math items. A break was inserted in between content areas, 
prompting participants to take a brief break if needed. 


In selecting the items, priority was given to selecting test items with high numbers of 
career-readiness features, as described in our prior report (Kao et al., 2018). Consideration was 
also given for logistics and feasibility of a paper-based administration. Early selections of test 
items were internally tested on undergraduate students for timing and feasibility. Results of 
these trial runs led to a few changes in item selections, including the omission of a constructed- 
response writing item which took too much time (an average of 22 minutes) to complete. 


We also considered Smarter Balanced content claims. Content claims are summary 
statements about the knowledge and skills students are expected to demonstrate on the 
assessment related to a particular aspect of the standards. For the present study, we 
considered the four Smarter Balanced ELA claims and the four Smarter Balanced math claims to 
ensure that our test instrument had adequate representation of each claim. However, items 
representing Smarter Balanced ELA Claim 3, Speaking and Listening, were not selected due to 
logistical and feasibility concerns for administration. Thus, the nine ELA items included in the 
present study consisted of the other three Smarter Balanced ELA claims (three Claim 1, three 
Claim 2, and three Claim 4 items). Selected math items represented each of the four math 
claims (three Claim 1, two Claim 2, two Claim 3, and two Claim 4). 


Table 2 provides detailed information for each of the items included in our test 
instrument, including number of career-readiness features, Smarter Balanced claims, Smarter 
Balanced item type, depth of knowledge?’ ratings, item response theory (IRT) item difficulty 
estimate (based on the b parameters from the Smarter Balanced operational IRT model), 
scoring, and mean performance from participants in the current sample. Each item was worth 
one point except for Math Item 9, which was worth two points, for a maximum total raw test 
score of 19 points. 


4Depth of knowledge is the level of cognitive demand or cognitive complexity required by a standard, target, or 
item, based on the work by Webb (1997). For more information related to Smarter Balanced items, please see 
www.smarterbalanced.org 
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Table 2 


Information on Smarter Balanced Claims, Item Type, Depth of Knowledge, Item Difficulty, Scoring, and Mean Performance From Present Study by 
Item Number 


Performance of 
sample in present 


study 
Item Number of career- Claim Item Depthof  Itemdifficulty Scoring (points 
Area number’ readiness features number Claimdescription type knowledge (IRT-B) possible) n M SD 
ELA 1 9 4 Research/Inquiry mc 2 0.61 1 298 .59 49 
2 5 2 Writing mc 2 0.68 1 297.72 45 
3 6 1 Reading ebsr 3 2.03 1 297 ~—«.54 50 
4 6 2 Writing ms 2 2.24 1 297 ~—s.38 49 
5 5 1 Reading htq 2 1.11 1 263.77 42 
6 7 1 Reading mc 4 1.89 1 297 42 49 
7 6 4 Research/Inquiry ms 2 2.05 1 295 42 50 
8 7 4 Research/Inquiry mc 2 1.00 1 297 53 50 
9 7 2 Writing mc 2 1.73 1 296.34 48 
Math 1 9 1 Concepts & mc 2 0.09 1 298 74 44 
Procedures 
2 10 2 Problem Solving eq 2 2.79 1 285 ~=.27 45 
3 10 1 Concepts & gi 2 2.56 1 292 .32 .47 
Procedures 
4 12 4 Modeling & Data eq 3 2.33 1 281 #44 50 
Analysis 
5 11 1 Concepts & mi 2 2.73 1 288 .26 44 
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Number of career- 
readiness features 


Item 
Claim description type 


Performance of 
sample in present 
study 


Item difficulty Scoring (points 


Procedures 
Problem Solving eq 


Communicating mc 
Reasoning 


Modeling & Data eq 
Analysis 


Communicating gi 
Reasoning 


No 


275 ~—-«.30 .46 
287 — 71 .46 


266 =.18 39 


295 1.04 0.87 
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Note. Smarter Balanced item type abbreviations: ebsr = evidence-based selected response; eq = equation or numeric; gi = grid item; htq = hot text; mc = 
multiple choice; ms = multiple select. Depth of Knowledge ratings: 1 = recall and reproduction; 2 = basic skills and concepts; 3 = strategic thinking and 
reasoning; 4 = extended thinking. For more information on Smarter Balanced test items, please visit www.smarterbalanced.org 


For a paper-based administration, some minor modifications from the computer-based 
items were necessary. For instance, ELA Item 5 was a “hot text” (htq) item in which the test 
taker must “mouse over” a paragraph to find the clickable answer choices (in this case, a single 
word) and select one answer. In our paper version, we highlighted the three answer choices 
and asked participants to circle the correct word. Among the math items, “equation” (eq) type 
items were modified to add a blank line or box for participants to handwrite their final answers 
(original items provided a numeric keypad for test takers to click). “Grid item” (gi) type items 
required minor layout changes, and wording changes from “select” to “circle.” All other items 
were printed in their original formats to the extent possible. Each item spanned one or two 
pages of the test booklet. The test was printed double-sided in booklet format with saddle 
stitching. 


Table 3 shows the career-readiness features rated (as described in Kao et al., 2018) for 
each of the 18 items selected for the present study. A total of 19 key career-readiness features 
were represented across the 18 test items. Each item contained between five and 12 features, 
with an average of 8.4. ELA items contained an average of 6.4 features while math items 
contained an average of 10.4 features. 
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Table 3 


Career-Readiness Features by Test Item 


ELA item number 


Math item number 


Feature 1 3 4 5 6 8 1 2 3 4 5 6 7 8 9 Total 

Reading comprehension X X XK X xX X Xx 'Wh %X® XK KX KX KX XK X 18 
Deductive reasoning X X xX X X X 'h® X XK XK XK KR XK X 17 
Importance of being exact or accurate X X XK X xX X X 'h XK RK XK X X xX 17 
Inductive reasoning X X X X xX xX X Bh X X xX 11 
Time sharing X X XK X xX X X 10 
Written comprehension X Xx X xX xX X 9 
Number facility Xx 'Wh WRX XK X KX KX XK X 9 
Processing information X '%h® X XK KX KR XK XK X 9 
Mathematical reasoning X 'Wh %X® WRK KX KX XK X 8 
Analyzing data or information X X 'Wh %X& XK KX XK X 8 
Making decisions and solving problems X X X X 6 
Complex problem solving xX xX X XxX xX X 6 
Identifying objects, actions, or events X X X xX xX 5 
Flexibility of closure X X X 4 
Getting information X X X X 4 
Organizing, planning, and prioritizing work xX xX X X 4 
Estimating the quantifiable characteristics X xX xX 3 
Critical thinking X X 2 
Active learning X X 2 
Total 9 6 6 5 7 7 9 10 10 12 11 10 10 11 11 
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Table 4 shows the definitions of the 19 career-readiness features (in alphabetical order) 
that were found in the items investigated in the present study, along with respective O*NET 
importance ratings by career domain (EMT and web developer). Importance ratings indicate the 
degree to which a descriptor is important to the occupation out of 100, and are based on 
survey responses collected by the U.S. Department of Labor. 
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Table 4 


Career-Readiness Feature Descriptions and O*NET Importance Ratings 


O*NET importance rating 


Feature Description EMT Web developer 
Active learning Understanding the implications of new 72 63 
information for both current and future 
problem-solving and decision-making. 
Analyzing data or Identifying the underlying principles, reasons, 64 64 
information or facts of information by breaking down 
information or data into separate parts. 
Complex problem solving Identifying complex problems and reviewing 63 66 
related information to develop and evaluate 
options and implement solutions. 
Critical thinking Using logic and reasoning to identify the 75 69 
strengths and weaknesses of alternative 
solutions, conclusions, or approaches to 
problems. 
Deductive reasoning The ability to apply general rules to specific 75 72 
problems to produce answers that make 
sense. 
Estimating the Estimating sizes, distances, and quantities; or 61 58 
quantifiable determining time, costs, resources, or 
characteristics materials needed to perform a work activity. 
Flexibility of closure The ability to identify or detect a known 56 53 
pattern, figure, object, word, or sound that is 
hidden in other distracting material. 
Getting information Observing, receiving, and otherwise 81 87 
obtaining information from all relevant 
sources. 
Identifying objects, Identifying information by categorizing, 81 67 
actions, or events estimating, recognizing differences or 
similarities, and detecting changes in 
circumstances or events. 
Importance of being exact Being very exact or highly accurate is 62° 42° 
or accurate important to performing this job. 
Inductive reasoning The ability to combine pieces of information 75 63 
to form general rules or conclusions. 
Making decisions and Analyzing information and evaluating results 88 81 


solving problems 


to choose the best solution and solve 
problems. 
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O*NET importance rating 


Feature Description EMT Web developer 
Mathematical reasoning The ability to choose the right mathematical 41 53 
methods or formulas to solve a problem. 

Number facility The ability to add, subtract, multiply, or 38 44 
divide quickly and correctly. 
Organizing, planning, Developing specific goals and plans to 68 73 
prioritizing work prioritize, organize, and accomplish your 
work. 
Processing information Compiling, coding, categorizing, calculating, 69 78 
tabulating, auditing, or verifying information 
or data. 
Reading comprehension — Understanding written sentences and 63 66 


paragraphs in work related documents. 


Time sharing The ability to shift back and forth between 53 22 
two or more activities or sources of 
information. 

Written comprehension — The ability to read and understand 63 66 


information and ideas presented in writing. 


“Importance of being exact or accurate refers to the percentage of respondents reporting it as “extremely 
important.” The other ratings indicate the degree to which a descriptor is important to the occupation out of 100. 
For more information on ratings, see www.onetonline.org 


Background surveys. Four surveys were developed: one for high school students, one for 
community college students, and two for workforce participants. All four background surveys 
began with a 10-item General Self-Efficacy (GSE) scale (Schwarzer & Jerusalem, 1995), which 
measures “perceived self-efficacy,” or the belief that one can perform novel or difficult tasks 
and facilitates goal setting, effort investment, persistence in the face of barriers, and recovery 
from setbacks. Samples from 23 nations (using translated versions) had Cronbach’s alphas 
ranging from .76 to .90. The purpose of including the GSE was to explore any potential 
noncognitive factors on test performance, with implications for career readiness. Additional 
detail on each background survey is described below. The surveys are included in Appendix B, 
Appendix C, Appendix D, and Appendix E. 


High School Survey 


In addition to the GSE, the high school background survey contained 13 items: six 
questions about education and career interests, and seven general demographic questions: 
age, grade level, gender, frequency of languages other than English spoken at home (as a proxy 
for English learner, or EL), number of computers or laptops at home (as a proxy for 
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socioeconomic status), race/ethnicity, and parent or guardians’ highest level of education (as a 
second proxy for socioeconomic status). See Appendix B for the survey. 


Community College Survey 


This survey included the GSE scale as well as 16 items querying interests and education, 
including name of their program, degrees obtained and to be obtained, length of program, 
course names, employment status, and job titles if any. Some items were used to ascertain 
participant eligibility. Four demographic questions were also included (age, gender, frequency 
of languages other than English spoken at home when growing up, and race/ethnicity). See 
Appendix C for the survey. 


Workforce EMT Survey 


This survey included the GSE scale along with 12 items querying education/training 
(including degrees held, EMT refresher course dates, current education program enrollment, 
and future education goals) and employment (including job status/hours worked, job titles, and 
management experience). Four demographic questions were also included (age, gender, 
frequency of languages other than English spoken at home when growing up, and 
race/ethnicity). See Appendix D for the survey. 


Workforce Web Developer Survey 


In addition to the GSE, 16 items queried education/training (including degrees held, self- 
directed course descriptions, and future education goals) and employment (including job 
status/hours worked and job titles). Web developers were also asked to list their experience in 
managing a team, working individually, programming, web design, and back-end development, 
as well as rate their expertise for each. Four demographic questions were also included (age, 
gender, frequency of languages other than English spoken at home when growing up, and 
race/ethnicity). See Appendix E for the survey. 


Research Design 


This study used a quasi-experimental between-subjects design in order to explore 
differences in performance of test items exhibiting career-readiness features across three 
different subgroups: high school students, community college students, and workforce 
participants. 


Procedure 


Timeframe. Data collection was conducted between October 2017 and March 2018. High 
school data collection took place in October and November. Community college data collection 
took place primarily in December and January, while workforce data collection took place 
primarily in February and March. For high school participants, data collection was scheduled 
with deference to school scheduling and assessment concerns. For community college 
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participants, data collection was scheduled after the school semester began to help ensure 
participants met criteria for program enrollment. 


Location. All high school data collection occurred in high school classrooms. Students 
from both Schools A and E were regular after-school program participants and participated in 
our study during after-school hours. Students from both Schools B and D participated during 
regular classroom hours. Data collection for the other subgroups occurred either in 
participants’ classrooms, or participants came to our onsite research lab. More specifically, 
participants from Sites F, H, and J took the study in their respective classrooms. Participants 
from Site G participated either on their campus or at our onsite research lab. Participants from 
other sources participated in the study at our onsite research lab. 


Classroom-based data collection was coordinated with the help of the teachers or 
instructors. Participants who came to our research lab generally scheduled themselves through 
a web-based calendar or were referred to us by the employment staffing agency described 
earlier. The web-based calendar was accessed through an online screening survey which 
filtered out those who did not meet the basic criteria for participation. 


Administration. The test instrument and background survey were designed to be 
administered together in approximately one hour. Participants were first introduced to the 
study and then gave informed consent. They were asked to sign a nondisclosure agreement per 
Smarter Balanced, and then asked to silence their cellular phones and place them in individual, 
clear plastic pencil boxes for added security. High school students under age 18 also returned 
signed parent permission forms prior to test administration. Participants were administered the 
test first, then the background survey. Participants randomly received either test Form E or 
Form M.° Completed test booklets were collected before administering the survey. Participants 
were allowed to write anywhere in the test booklet and marked answers directly into the test 
booklet in either pen or pencil. No calculators were allowed. After completion of the survey, 
participants were given a gift card and their phones were returned. 


Incentives. High school students were given a $25 Visa gift card, while community college 
students were given a $40 Amazon gift card, and workforce participants were given a $65 
Amazon gift card. The onsite participants were also provided paid parking, if needed. 


Data entry and scoring. Raw responses from the test instrument and the four background 
surveys were entered in pairs, with one researcher reading aloud and a second researcher 
electronically entering. A third researcher entered a random selection of 20% as a reliability 
check, with over 98% agreement for each instrument. Any errors found were corrected. The 
test instrument contained one open-ended response (Math Item 6, an equation), which was 
scored by three researchers with 94.1% interrater agreement. All other test items were 
converted into scores using statistical software. 


5Roughly half of the participants within each subgroup received Form E, while the other half received Form M. 
Independent t-test results showed no significant differences between the two forms on overall test performance. 
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Using multiple-group item response theory (IRT) analysis with the two-parameter logistic 
(2PL) model (Smarter Balanced’s operational IRT model), we applied Smarter Balanced item 
parameters from the calibrated item pool to create estimated IRT scaled scores for additional 
analyses. The 2PL model’s item parameters were constrained to their calibrated values from 
the operational item pool. The population means and variances were freely estimated from the 
item response data. flexMIRT® (Houts & Cai, 2016) was used for the IRT analyses. Expected a 
posteriori (EAP) scaled scores were estimated. We note that we do not seek to replicate 
Smarter Balanced’s operational scoring approach because the items were purposefully selected 
and the sampling was not random. EAP scores are Bayesian estimates that have more superior 
statistical properties when the number of items is not large (Thissen & Wainer, 2001). 


Data analysis. Descriptive statistics and correlations were conducted on the test data and 
the survey data. Analyses of variance and regression modeling were conducted on the test 
scores (both raw scores and IRT scaled scores) in order to explore differences across the 
subgroups. 


Results 


Detailed results from the background surveys are presented in Appendix F. These include 
General Self-Efficacy scores and additional demographic information for each of the three 
subgroups, and education, training, and work experience for the community college and 
workforce subgroups. 


As described in the Method section, the maximum raw score on our paper-based test was 
19 (9 points for ELA and 10 points for math). Table 5 shows the descriptive results of the raw 
test scores by subgroup and by content area. 
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Table 5 
Descriptive Results of the Raw Test Scores by Subgroup and by Content Area 


Subgroup Content area n Minimum Maximum M SD 
High school ELA 103 0 8 3.90 1.77 
Math 103 0 9 3.26 2.24 

Total 103 0 17 7.17 3.52 

Community college ELA 111 0 9 4.75 1.81 
Math 111 0 10 3.98 2.57 

Total 111 1 18 8.73 3.77 

Workforce ELA 84 1 9 5.29 1.89 
Math 84 0 10 5.37 2.65 

Total 84 2 18 10.65 4.10 


As described in the Method section, IRT scores were created by applying Smarter 
Balanced item parameters to the raw scores. Table 6 shows descriptive results of the IRT scaled 
scores. 


Table 6 
Descriptive Results of IRT Scores by Subgroup and by Content Area 


Subgroup Content area n Minimum Maximum M SD 
High school ELA 103 0.047 2.720 1.266 0.623 
Math 103 -0.375 3.263 0.999 0.941 
Total 103 -0.734 3.349 1.114 0.857 
Community college ELA 111 0.474 3.255 1.878 0.669 
Math 111 -0.322 4.142 1.507 1.140 
Total 111 0.012 3.677 1.724 0.790 
Workforce ELA 84 0.748 3.668 2.318 0.754 
Math 84 0.351 4.210 2.380 1.005 
Total 84 0.339 4.420 2.354 0.978 


Analyses of variance and regression analyses were conducted on the raw test scores as 
well as the IRT scores. Overall findings were similar, however we report results from the 
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analyses conducted on IRT scores because they are more sensitive in detecting group 
differences. 


In a one-way analysis of variance conducted on the overall total score, results indicated 
significant differences between the subgroups on the total test score, F(2, 295) = 47.19, 
p<.001. Post hoc comparisons using Tukey HSD indicated that the workforce subgroup 
performed significantly higher than the community college subgroup (p < .001) and significantly 
higher than the high school subgroup (p < .001). The community college subgroup also 
performed significantly higher than the high school subgroup (p < .001). 


On the ELA score, a one-way analysis of variance showed significant differences between 
the subgroups, F(2, 295) = 57.00, p < .001. Post hoc comparisons using Tukey HSD indicated that 
the workforce subgroup performed significantly higher than the community college subgroup 
(p < .001) and significantly higher than the high school subgroup (p < .001). The community 
college subgroup also performed significantly higher than the high school subgroup (p < .001). 


On the math score, a one-way analysis of variance also showed significant differences 
between the subgroups, F(2, 295) = 41.41, p < .001. Post hoc comparisons using Tukey HSD 
indicated that the workforce subgroup performed significantly higher than the community 
college subgroup (p < .001) and significantly higher than the high school subgroup (p < .001). 
The community college subgroup also performed significantly higher than the high school 
subgroup (p < .01). 


Table 7 shows regression results, comparing the three subgroups on the IRT scores for 
ELA, math, and the total. High school was designated as the reference group. 
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Table 7 


Results From Regressions Comparing Subgroup Performance on IRT Scores by 


Content Area 

Area Model B SE B t p 

ELA = (Constant) 1.266 .067 18.926 <.001 
CC vs. HS 0.612 .093 372 6.587 <.001 
WF vs. HS 1.052 .100 595 10.539 <.001 

Math (Constant) 0.999 .102 9.782 <.001 
CC vs. HS 0.507 .142 .210 3.575 <.001 
WF vs. HS 1.380 .152 532 9.055 <.001 

Total (Constant) 1.114 .086 13.001 <.001 
CC vs. HS 0.610 .119 .297 5.128 <.001 
WF vs. HS 1.240 .128 561 9.702 <.001 


Note. HS = high school, CC = community college, WF = workforce. R square for 
ELA = .28; R square for Math = .22, R square for Total = .24. 


Career Domains 


Additional analyses were conducted on the two different career domains. Table 8 and 
Table 9 show the means and standard deviations of the raw test scores and the IRT scores, 
respectively, by career domain and by subgroup. 


Table 8 


Raw Test Score Means (SD) by Career Domain and by Subgroup 


EMT Web developer 
High school Community college Workforce Community college Workforce 
Area (n = 103) (n = 64) (n = 45) (n = 47) (n = 39) 
ELA 3.90 (1.77) 4.77 (2.02) 4.71 (1.85) 4.72 (1.50) 5.95 (1.72) 
Math 3.26 (2.24) 3.89 (2.82) 4.24 (2.51) 4.11 (2.20) 6.67 (2.20) 
Total 7.17 (3.52) 8.66 (4.15) 8.96 (3.95) 8.83 (3.23) 12.62 (3.37) 
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Table 9 
IRT Score Means (SD) by Career Domain and by Subgroup 


EMT Web developer 
High school Community college Workforce Community college Workforce 
Area (n = 103) (n = 64) (n = 45) (n = 47) (n = 39) 
ELA 1.266 (0.623) 1.868 (0.736) 2.072 (0.741) 1.892 (0.574) 2.603 (0.672) 
Math 0.999 (0.941) 1.432 (1.223) 1.958 (0.945) 1.608 (1.020) 2.866 (0.849) 
Total 1.114 (0.857) 1.689 (0.865) 1.941 (0.940) 1.770 (0.680) 2.830 (0.794) 


EMT. A one-way analysis of variance on total IRT score was conducted comparing the 
performance of high school students and community college EMT students and workforce EMTs 
(i.e., web developers were excluded from the analyses). Results were significant, 

F(2, 209) = 16.94, p < .001. Post hoc comparisons using Tukey HSD revealed significant 
differences in performance between high school students and community college EMT students 
(p < .001) and between high school students and workforce EMTs (p < .001). No significant 
differences were detected between community college EMTs and workforce EMTs on total test 
score. 


A similar pattern of performance was found on the ELA score. A one-way analysis of 
variance showed significant differences between the subgroups, F(2, 209) = 27.79, p< .001. 
Post hoc comparisons using Tukey HSD revealed significant differences in performance between 
high school students and community college EMT students (p < .001) and between high school 
students and workforce EMTs (p < .001). No significant differences were detected between 
community college EMTs and workforce EMTs on the ELA score. 


On the math score, a one-way analysis of variance also showed significant differences 
between the subgroups, F(2, 209) = 13.85, p < .001. Post hoc comparisons using Tukey HSD 
indicated that the workforce subgroup performed significantly higher than the community 
college subgroup (p < .05) and significantly higher than the high school subgroup (p < .001). The 
community college subgroup also performed significantly higher than the high school subgroup 
(p < .05). 


Web developer. A one-way analysis of variance on total IRT score was conducted 
comparing the performance of high school students and community college web developer 
students and workforce web developers. Results were significant, F(2, 186) = 65.46, p < .001. 
Post hoc comparisons using Tukey HSD indicated that the workforce web developers performed 
significantly higher than the community college web developer students (p < .001) and the high 
school students (p < .001). Community college web developer students also performed 
significantly higher than the high school students (p < .001). 
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A similar pattern of performance was found on the ELA score. A one-way analysis of 
variance showed significant differences between the subgroups, F(2, 186) = 68.46, p < .001. 
Post hoc comparisons using Tukey HSD indicated that the workforce web developers performed 
significantly higher than the community college web developer students (p < .001) and the high 
school students (p < .001). Community college web developer students also performed 
significantly higher than the high school students (p < .001). 


One-way analysis of variance on the math score also showed significant differences 
between the subgroups, F(2, 186) = 55.56, p < .001. Post hoc comparisons using Tukey HSD 
indicated that the workforce subgroup performed significantly higher than the community 
college subgroup (p < .001) and significantly higher than the high school subgroup (p < .001). 
The community college subgroup also performed significantly higher than the high school 
subgroup (p < .01). 


Bachelor’s Degrees 


Based on background survey responses, we found that many of the non-high school 
participants already possessed bachelor’s degrees or higher, which suggests potential 
education and training beyond what is needed to become an EMT or a web developer. Since 
our exemplar careers were initially selected because they do not require a bachelor’s degree, 
we conducted further exploration into participants with such degrees. 


There was a moderate positive correlation between the total IRT score and (any) 
bachelor’s degree (and higher), r(297) = .44, p < .001. There was also a slight positive 
correlation between the total test score and participant age, r(294) = .32, p < .001. Participant 
age was positively correlated with bachelor’s degree, r(294) = .60, p < .001. 


Table 10 and Table 11 show the raw test score means and IRT score means, respectively, 
of participants with and without bachelor’s degrees or higher® by subgroup and by career 
domain. 


6Among community college participants, two EMT students and two web development students also held degrees 
above a bachelor’s. Among the workforce participants, six web developers held degrees above a bachelor’s. No 
workforce EMTs held degrees above a bachelor’s. 
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Table 10 


Raw Test Score Means of Participants With and Without Bachelor’s Degrees or Higher by Subgroup and 
by Career Domain 


Community college Workforce 
Bachelor’s + 
Career domain degrees n M SD n M SD 
EMT No 44 7.80 3.94 35 8.34 3.61 
Yes 19 10.47 4.14 10 11.10 4.53 
Web developer No 29 8.07 3.13 7 13.00 4.00 
Yes 18 10.06 3.10 32 12.53 3.28 
Total No 73 7.90 3.62 42 9.12 4.03 
Yes 37 10.27 3.63 42 12.19 3.61 


Note. There was one missing case for community college EMT. 


Table 11 


IRT Score Means of Participants With and Without Bachelor’s Degrees or Higher by Subgroup and by 
Career Domain 


Community college Workforce 
Bachelor’s + 
Career domain degrees n M SD n M SD 
EMT No 44 1.516 0.837 35 1.784 0.850 
Yes 19 2.072 0.845 10 2.488 1.079 
Web developer No 29 1.601 0.670 7 2.990 1.064 
Yes 18 2.043 0.618 32 2.795 0.739 
Total No 73 1.550 0.771 42 1.985 0.986 
Yes 37 2.058 0.733 42 2.722 0.828 


Note. There was one missing case for community college EMT. 


Analysis of variance after excluding participants with bachelor’s degrees (n = 218 [103 
high school, 73 community college, 42 workforce]) was conducted on the total IRT score, which 
showed significant differences between the subgroups F(2, 215) = 16.57, p < .001. Post hoc 
comparisons using Tukey HSD indicated that the workforce subgroup performed significantly 
higher than the community college subgroup (p < .05) and significantly higher than the high 
school subgroup (p < .001). The community college subgroup also performed significantly 
higher than the high school subgroup (p < .01). This pattern of results is similar to the pattern of 
results for the overall sample. 
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Exploring Background Factors Within Subgroups 


In addition to differences across the three subgroups, there may be differences within 
each subgroup that play a role in the performance of our test instrument. Such differences are 
worth preliminary exploration in spite of small sample sizes to inform future research. 


High school participants. As described in the Method section, our sample of high school 
students consisted of 52.9% female and 47.1% male participants, which is roughly consistent 
with the national pattern. Female high school students scored significantly lower (n = 54, 

M = 6.44, SD = 3.24) on our test instrument than male students (n = 48, M = 8.02, SD = 3.68), 
t(100) = 2.30, p < .05. This is different from the general trend of student performance in the 
state of California for Grade 11, in which during Spring 2017 testing, 65% of females and 54% of 
males met or exceeded standards in ELA, and 33% of females and 32% of males met or 
exceeded standards in math (California Department of Education, 2017). In our sample, no 
significant differences were found between females and males on ELA items, while female 
participants performed significantly lower on the math items (M = 2.80, SD = 1.99) than their 
male peers, (M = 3.83, SD = 2.39), t(100) = 2.39, p< .05. 


There were no significant correlations between total test score, General Self-Efficacy 
score, high school site, or EL proxy. Students who reported never speaking a language other 
than English at home were more likely to have a higher General Self-Efficacy score. 


Community college participants. There were no significant correlations between total 
test score, General Self-Efficacy score, or gender for our community college participants. This 
was true within both career domains. 


Workforce participants. Among the 45 workforce EMT participants, 25 reported being 
currently enrolled in a program to become a paramedic. Those who reported being enrolled in 
a paramedic program performed significantly lower (M = 6.36, SD = 2.45) than those who did 
not (M = 12.20, SD = 2.93), t(43) = 7.29, p < .001. As described in our Method section, 24 of 
these 25 paramedic students were from the same source and participated in the study at the 
same time, on the evening of the second day of an intensive, full-time program. Admission to 
this program requires current EMT certification and over 1,200 hours of experience working as 
an EMT. The instructor and several participants (during informal conversations) reported that 
the students had all quit their jobs as EMTs within the past couple of weeks. However, due to 
the unexpected participant recruitment source, the exact length of time that had elapsed since 
quitting was not captured in our background survey. 


There were no significant correlations between total test score, General Self-Efficacy 
score, participant age, gender, length of work experience, or experience managing a team for 
our workforce EMT participants. 


Among the workforce web developers, there was a moderate negative correlation with 
the total test score and reported age (measured in years), r(39) = -.37, p < .05. As reported in 
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the Method section, the age range of the workforce web developers in this study was 20 to 50, 
with a mean of 33.8 (SD = 9.8), and a median of 32.0. Female participants performed 
significantly higher (n = 10, M = 14.60, SD = 2.32) than male participants (n = 29, M = 11.93, 

SD = 3.43), t(37) = -2.28, p < .05. This trend is opposite from the trend for our high school 
participants. 


There were no significant correlations between the total test score, General Self-Efficacy 
score, employment status (i.e., those employed 20 or more hours a week vs. those employed 
less than 20 hours a week), length of work experience, or experience managing a team for our 
workforce web developers. 


Discussion 


The purpose of this study was to serve as a preliminary validation of the career-readiness 
features found in a targeted set of Grade 11 assessment items by exploring performance of 
participants with varying levels of experience in two exemplar careers (EMT and web 
developer). As expected, results indicated that workforce individuals scored significantly higher 
on the test than community college students, who in turn also scored significantly higher than 
high school students. Results suggest that having added experience in a field leads to refining 
certain career-readiness skills found in high-school level content-area assessments. 


The quasi-experimental nature of the design, along with the small sample size of this 
study, begs caution in the interpretation of results. We recognize that the three subgroups have 
inherent differences beyond simply “career experience” that may never be captured, even ina 
controlled study. For instance, it may not be possible to fully disentangle the effects of age and 
experience. Some skills, such as deductive and inductive reasoning, may be honed through 
other life experiences. Others are honed in college, regardless of the field of study. Further, it is 
possible that there may be some skew within the community college group, as some lower 
scorers may also never complete their degree and join the workforce in the given domain, but 
impact current scores. Our sample consisted of a high number of participants who held 
bachelor’s degrees and above, including community college students, who may have returned 
to school for a career change. Coincidentally, two web developer students (who were excluded 
from the final analyses) had prior careers as EMTs, reflecting the general transient nature of 
some careers, especially ones that do not require extensive education or training. Our initial 
goals in selecting these two exemplar careers were precisely because they did not require four- 
year degrees, but perhaps the reality of the 21° century is that obtaining a bachelor’s helps 
young adults with the knowledge, skills, and attributes necessary to remain competitive in the 
market. 


Touched upon briefly in our background survey (see Table F19 in Appendix F) was the 
motivation for choosing a career. Not surprisingly, a high number of those who selected 
“salary” were web developers while a high number of those who selected “help people” were 
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EMTs, especially given that the median salary of web developers is roughly double that of 
EMTs. Those motivated to have a career that earns more may also be more motivated to obtain 
a four-year degree. Though only a few EMTs in our sample held bachelor’s degrees, those who 
did scored higher on our test. In general, participants with bachelor’s degrees scored higher 
than those without. After excluding participants with bachelor’s degrees, the pattern of results 
was similar to the whole sample, in that workforce individuals scored significantly higher than 
community college students, who in turn scored significantly higher than high school students. 
This strengthens the suggestion that added work experience can lead to performing higher on 
test items found to contain many career-readiness features. 


In exploring background factors, we found, surprisingly, no significant relationships 
between self-efficacy and test performance. However, EMTs were more likely to report higher 
self-efficacy than web developers—perhaps important in a career where salary is low but 
motivation to “help people” is high. 


Among workforce web developers, there was a moderate negative correlation of test 
performance with age, which suggests that an increase in age does not necessarily mean 
greater career-readiness skills. Skills needed to “be ready” do not mean skills needed to 
continue in or excel at a career. Those would be beyond the scope of the present study. 


Finally, a notable finding was the trend of female workforce web developers performing 
significantly higher than male workforce web developers, despite being underrepresented in 
this field in both our sample and in the general population. This trend is opposite from the 
trend of our female high school participants (who performed significantly worse than their male 
peers). Though the sample size was small, it leads to other questions that might be explored in 
future work. Do women pursuing careers as web developers have to be “more ready” at 
needed skills in order to be hired or to be successful? Or are women who excel at math more 
likely to consider web development as a career? Such questions remain unanswered for the 
moment. 


Limitations and Future Work 


This study has several limitations. As discussed earlier, there are inherent differences 
between the three subgroups beyond career experience. It would be challenging to completely 
isolate the effect of career experience in a true experiment, as participants have varying life 
experiences that confound the effect of work experience. In addition, while the initial goal of 
the study was to compare participant subgroups across both career domains, the two career 
domains invariably have differences that cannot be ignored. One is commonly viewed as a 
“stepping stone” job toward other careers, and the other is more likely to be a long-term 
career. A study on careers that are more similar to each other in training, median salary, long- 
term prospects, etc. may have yielded different results. 


Web developer, as a career, is also less well-defined than EMT. EMTs take very specific 
courses that follow national standards and must carry a license. Web development is broad and 
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encompasses a variety of tasks and activities that fall under the umbrella term. Our efforts to 
narrow criteria for recruitment were met with challenges in reaching our targeted N, and our 
final sample included participants with varying training, skill sets, and experience—at least in 
contrast with the more well-defined EMT. Overall, EMTs were also younger than web 
developers. EMTs were also typically working under 40 hours a week, which reflects the nature 
of the field. 


Nearly half of our workforce EMT sample came from one source—the second day of a 
community college-based paramedic program. As reported in the Results section, these 
participants performed significantly lower than the other workforce EMTs, and the workforce 
EMTs overall did not perform significantly different from the community college EMTs. This was 
a self-selected group who all elected to quit their jobs as EMTs and advance into the “next 
step” as paramedics. There may be differences between those who choose to become 
paramedics and those who do not. Those who do not may have medical school or other 
pursuits as an end goal. Or this may be related to the community and the type of students 
attracted to such a program. Or it may be because this was an intensive, full-time program that 
began early in the morning, and we were conducting the study in the evening, after a long day, 
and participants were mentally strained. Ideally, our sample should have represented a greater 
variety of sources, which may have yielded different results. 


Although we began with a set of 36 career-readiness features at the outset, 19 were 
represented in the present study. Remaining features should be reconsidered in a validation 
study. Additionally, while we selected test items with high numbers of features for 
examination, it is important to note that the features are not necessarily mutually exclusive. As 
shown in our companion report (Kao et al., 2018), some features were significantly correlated 
with one another, depending on the content area. Reading comprehension in math, for 
example, was moderately correlated with analyzing data or information. In ELA, critical thinking 
was moderately correlated with making decisions and solving problems. Future work should 
investigate these relationships further and consider the weight each career-readiness feature 
has within a test item. 


There is a general self-selection bias in such research studies, in that all participants were 
aware of the gift card incentive during the recruitment process. Those not swayed by the 
incentive may feel the need to be of help to research. These factors are worth mentioning, but 
are beyond the scope of the present study. 


There are also limitations to our test instrument. Ultimately, the test items were adapted 
from a computer-based test, and administered in a setting for which they were not designed. In 
the computer-adaptive test, test takers are given items based on their answers to the previous 
item. This was not possible for the present study. Further, while the decisions we made on the 
test items were based on the presence of career-readiness features, it is not possible to 
separate the confounding effect of content knowledge. Mathematics, especially, can be 
challenging for individuals who have not recently taken a course in mathematics (and 
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challenging even for those who have). English language arts may have less of a confounding 
effect in this regard. However, those far removed from standardized test taking (due to age or 
alternate or international schooling) have their own unique issues and barriers. 


Future work might consider exploring features through computer-adaptive testing. This 
might also allow the exploration of a greater number of test items. Future work might also 
compare one exemplar career with a control group consisting of a variety of careers in order to 
identify specific features relevant to that career. Future feature analysis studies might also be 
devoted to more fast-growing female-dominated professions, such as dental assistants, 
dieticians, or cosmetologists. 


Despite limitations, results from this study provide information that experience in a field 
is associated with higher performance on test items that contain large numbers of career- 
readiness features. Test items measuring English language arts and mathematics do contain 
indicators of career readiness. Inferences about career readiness, as well as college readiness, 
can be drawn from test scores. Preparation for such assessments can hone these skills and thus 
help high school students prepare for college and/or career. 
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Appendix A: 
College/Career Indicator Performance Levels 


Criteria 


Prepared Level: Does the graduate meet at least 1 measure below? 


A. Career Technical Education (CTE) Pathway Completion plus one of the following criteria: 


o Smarter Balanced Summative Assessments: At least a Level 3 “Standard Met” on ELA or Mathematics and at 
least a Level 2 “Standard Nearly Met” in the other subject area 


o One semester/two quarters of Dual Enrollment with passing grade (Academic/CTE subjects) 
B. At least a Level 3 “Standard Met” on both ELA and Mathematics on Smarter Balanced Summative Assessments 


C. Completion of two semesters/three quarters of Dual Enrollment with a passing grade (Academic and/or CTE 


subjects) 
D. Passing Score on two Advanced Placement (AP) Exams or two International Baccalaureate (IB) Exams 


E. Completion of courses that meet the University of California (UC) a—g criteria plus one of the following criteria: 


o CTE Pathway completion — Smarter Balanced Summative Assessments: At least a Level 3 “Standard Met” on 
ELA or Mathematics and at least a Level 2 “Standard Nearly Met” in the other subject area 


o One semester/two quarters of Dual Enrollment with passing grade (Academic/CTE subjects) 


o Passing score on one AP Exam OR on one IB Exam 


Approaching Prepared Level - Does the graduate meet at least 1 measure below? 


A. High School Diploma and any one of the following: 
o CTE Pathway completion 


o Scored at least Level 2 "Standard Nearly Met" on both ELA and Mathematics Smarter Balanced Summative 
Assessments 


o Completion of one semester/two quarters of Dual Enrollment with passing grade (Academic/CTE subjects) 


o Completion of courses that meet the UC or CSU a-g criteria 
Not Prepared Level 


Student did not meet any measure above or did not graduate, so considered NOT PREPARED 


Note. From “College/Career Indicator,” by The California Department of Education, 2018 


(https://www.cde.ca.gov/ta/ac/cm/cci.asp). 
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Appendix B: 
High School Background Survey 


Please check one of the four boxes for each item below. 


1. 1can always manage to solve difficult problems if I try hard enough. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


2. If someone opposes me, I can find the means and ways to get what I want. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


3. It is easy for me to stick to my aims and accomplish my goals. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


4.1 am confident that I could deal efficiently with unexpected events. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


5. Thanks to my resourcefulness, I know how to handle unforeseen situations. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


6. I can solve most problems if I invest the necessary effort. 


[_] Not at all true [_] Hardly true [_] Moderately true — [_] Exactly true 


7. can remain calm when facing difficulties because I can rely on my coping abilities. 


[_] Not at all true [_] Hardly true [_] Moderately true —_ [_] Exactly true 


8. When I am confronted with a problem, I can usually find several solutions. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 
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9. If 1am in trouble, I can usually think of a solution. 


[_] Not at all true [_] Hardly true [_] Moderately true [_] Exactly true 


10. I can usually handle whatever comes my way. 


[_] Not at all true [_] Hardly true [_] Moderately true [_] Exactly true 


Education & Interests 
1. Do you plan to pursue additional schooling after you graduate from high school? 


L_] No additional school [_] Vocational or certificate program 
|_] Community College [_] Four-year university 
[_] Other, please describe__ 


2. Which of the following jobs would you be interested in having, after you complete all 
education or training? (You can check more than one!) 


[_] Registered nurse [_] Tutor 

[_] Web administrator [_] Computer and information systems manager 
[_] Dental hygienist [_] Emergency Medical Technician (EMT) 

[_] Biomedical engineer [_] Video game designer 

[_] Genetic counselor [_] Music therapist 

[_] Physical therapist [_] Spa manager 

[_] Web developer [_] Climate change analyst 

L_] Medical assistant [_] Computer systems analyst 


[| Other, please describe 


3. What are the most important reasons why you might choose a specific career? (Check up to 
three!) 


[_] Interest in field [_] Less training or additional school needed 
[_] Location [_] Rapid growth/number of jobs available 
[_] Salary/compensation |_] Motivated to help society/people 


[| Other, please describe 
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4. What kinds of grades do you usually get in your classes? Please check one. 


[_] Mostly A’s 
[_] A’s and B’s 
L_] Mostly B’s 
[_] B’s and C’s 
[_] Mostly C’s or lower 


5. What kinds of grades do you usually get in your English or language arts classes? Please 
check one. 


[_] Mostly A’s 
[_] A’s and B’s 
L_] Mostly B’s 
[_] B’s and C’s 
[_] Mostly C’s or lower 


6. What kinds of grades do you usually get in your math classes? Please check one. 


[_] Mostly A’s 
[_] A’s and B’s 
L_] Mostly B’s 
[_] B’s and C’s 
[_] Mostly C’s or lower 


Background 


1. How old are you in years? 


[ ]16 mil [ ]18 [_] 19 or older 


2. Grade: [_]11th [_] 12th 


3. Gender: 
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4. How often do people in your home talk to each other in a language other than English? 


[_] Never [_] Once in a while [_] About half of the [_] All or most of the 
time time 


5. How many computers or laptops are in your home? (Only count the ones that are working) 


[ ]0 []1 [ ]2 [ ]3 [_] 4 or more 


6. With which of the following groups do you most strongly identify? (It’s okay to pick more 
than one!) 


[-] American Indian or Alaska Native [_] Hispanic / Latino/a 
[_] Asian or Pacific Islander [_] White or Caucasian 


[_] Black or African American [1] Other, please describe 


7. What is the highest level of education your parents or guardians completed? Please choose a 
box for each of the parents/guardians you live with. 


If you live with more than two parents/guardians, think about the two you spend the most time 
living with. If you live with only one parent/guardian, you can leave the second column blank. 
If you are not sure, please check the box that says “Not sure/don’t know.” 


Parent or Guardian #1: Parent or Guardian #2: 

[_] Less than high school [_] Less than high school 

[_] High school graduate or GED (equivalent) [_] High school graduate or GED (equivalent) 
[_] Some college (no degree) [_] Some college (no degree) 

[_] Associate’s or 2-year college degree [_] Associate’s or 2-year college degree 

[_] Bachelor’s or 4-year college degree [_] Bachelor’s or 4-year college degree 

[_] Master’s degree [_] Master’s degree 


[_] Advanced/professional degree (example: [_] Advanced/professional degree (example: 
MD) MD) 
[_] Not sure/don’t know [_] Not sure/don’t know 

Thank you! 
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Appendix C: 
Community College Background Survey 


Please check one of the four boxes for each item below. 


1. 1can always manage to solve difficult problems if I try hard enough. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


2. If someone opposes me, I can find the means and ways to get what I want. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


3. It is easy for me to stick to my aims and accomplish my goals. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


4.1 am confident that I could deal efficiently with unexpected events. 


[_] Not at all true [_] Hardly true [_] Moderately true —[_] Exactly true 


5. Thanks to my resourcefulness, I know how to handle unforeseen situations. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


6. I can solve most problems if I invest the necessary effort. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


7. can remain calm when facing difficulties because I can rely on my coping abilities. 


[_] Not at all true [_] Hardly true [_] Moderately true — [_] Exactly true 


8. When I am confronted with a problem, I can usually find several solutions. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 
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9. If Lam in trouble, I can usually think of a solution. 


[_] Not at all true [_] Hardly true [_] Moderately true [_] Exactly true 


10. I can usually handle whatever comes my way. 


[_] Not at all true [_] Hardly true [_] Moderately true [_] Exactly true 


Interests and Education 
1. In what field or program are you currently pursuing a degree or certification? 


[_] Emergency Medical Technician (EMT) [_] Web development 


[_] Other (please describe): 


2. What are the most important reasons you are interested in the job area you chose? (Please 
check up to three reasons) 


[_] Interest in field [_] Less training or additional school needed 
[_] Location [_] Rapid growth/number of jobs available 
[_] Salary/compensation [_] Motivated to help society/people 


[_] Other, please describe 


3. What kind of degree or certificate will you receive when you complete the program that you 
are currently enrolled in? Please specify. 


[_] Certificate, in: 


[_] Associate’s degree, in: 
[_] Other, please describe: 
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4. How long in total is your program? Please list in terms of years and months, or semesters, if 
applicable: 


years, months OR semesters OR hours 


5. How far along are you in the program? 


years, months OR semesters OR hours 


6. Are your courses conducted online or in person? 


[_] Online only [_] In person only [_] Both online and in person 


7. Please list the courses you have completed relevant to your specific program of study: 


8. Do you plan to pursue additional schooling after you finish your current program? 


[_] No additional school [_] Vocational or certificate program 
[_] Community college [_] Four-year university 
[_] Other, please describe__ 


9. Do you hold degrees or certificates in other fields? If yes, please check all that apply and 
describe: 
[_] No, no other degrees or certificates 
_] Yes, high school diploma or GED 


[_] Yes, certificate in: 


[_] Yes, two-year or Associate’s degree in: 


[_] Yes, four-year or Bachelor’s degree in: 
[_] Yes, other (please describe): 
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10. Are you currently enrolled in a program other than EMT or web development? 
(For example: Certificate in Accounting, Associate’s in Nursing, Bachelor’s in Anthropology, etc.) 


Employment 


1. Which of the following categories best describes your current employment status? 


[_] Employed, working 40 or more hours per [_] Not employed, looking for work 
week 

[_] Employed, working 20-39 hours per week [_] Not employed, NOT looking for work 
[_] Employed, working 1-19 hours per week [_] Other (please describe): 


2. Are you currently employed in a field related to your program of study? 


[_] Yes, job title: 
[_] No, not currently employed 
[_] No, currently employed in (please describe): 


Background 


1. How old are you in years? 


2. Gender: 


3. Growing up, how often did people in your home talk to each other in a language other than 
English? 


[_] Never [_] Once in a while [_] About half of the time [_] All or most of the time 


4. With which of the following groups do you most strongly identify? (It’s okay to pick more 
than one!) 


[_] American Indian or Alaska Native [_] Hispanic / Latino/a 


[_] Asian or Pacific Islander [_] White or Caucasian 
[_] Black or African American [1] Other, please describe 
Thank you! 
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Appendix D: 
EMT Background Survey 


Please check one of the four boxes for each item below. 


1. 1can always manage to solve difficult problems if I try hard enough. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


2. If someone opposes me, I can find the means and ways to get what I want. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


3. It is easy for me to stick to my aims and accomplish my goals. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


4.1 am confident that I could deal efficiently with unexpected events. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


5. Thanks to my resourcefulness, I know how to handle unforeseen situations. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


6. I can solve most problems if I invest the necessary effort. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


7. can remain calm when facing difficulties because I can rely on my coping abilities. 


[_] Not at all true [_] Hardly true [_] Moderately true — [_] Exactly true 


8. When I am confronted with a problem, I can usually find several solutions. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 
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9. If 1am in trouble, I can usually think of a solution. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


10. I can usually handle whatever comes my way. 


[_] Not at all true [_] Hardly true [_] Moderately true [_] Exactly true 


Education 
1. What degrees or certificates do you hold? (Check all that apply) 


[_] High school diploma or GED 
[_] Certificate in: 


[_] Two-year or Associate’s degree in: 


[_] Four-year or Bachelor’s degree in: 


[_] Master’s degree in: 
[_] Other, please describe: 
[_] Not applicable 


2. Do you feel that your education prepared you for your current employment? 


[_] Yes [_] No [_] Mixed 


Please explain why: 


3. When was the last time you completed continuing education courses to renew your EMT 
license? If this is not applicable, please write N/A. 


Date (Month/Year): 


4. Other than continuing education courses, are you currently enrolled in an educational 
program? (For example: Paramedic course, Associate’s in Nursing, Bachelor’s in 
Anthropology, etc.) 
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5. Do you plan to pursue any additional schooling? (Not including continuing education 


requirements). 
L_] No additional school [_] Vocational or certificate program 
[_] Community college [_] Four-year university 
[_] Other, please describe__ 
Employment 


1. Which of the following categories best describes your current employment status, as an 
EMT? 


[_] Employed, working 40 or more hours per week [_] Not employed, looking for work 
[_] Employed, working 20-39 hours per week [_] Not employed, NOT looking for work 
_] Employed, working 1-19 hours per week [_] Other (please describe): 


2. What is your current job title? If you are not currently employed, what was your most recent 


job title? 


3. Do you currently have other occupations aside from EMT? If yes, please list them below and 
write the number of hours per week for each occupation. If no, please write N/A. 


Other Current Occupations: Number of hours per week: 


4, What are the most important reasons you are interested in the job area you chose? (Please 
check up to three reasons) 


[_] Interest in field [_] Less training or additional school needed 
[_] Location [_] Rapid growth/number of jobs available 
[_] Salary/compensation [_] Motivated to help society/people 


[_] Other, please describe 
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5. How many years/months have you worked as an EMT? 


Years Months 


6. Have you ever managed a team of EMTs? If so, please indicate years and months spent 
managing a team: 


[_]No [_] Yes: Years Months 


7. What are your employment goals for the next few years? (Check all that apply) 


[_] Continue working as an EMT 

[_] Pursue a career as an advanced EMT or paramedic 
[_] Pursue other healthcare careers 

[_] Pursue a career NOT related to healthcare 

[_] Other, please describe:_ 
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Background 


1. How old are you in years? 


2. Gender: 


3. Growing up, how often did people in your home talk to each other 
in a language other than English? 


[_] Never [_] Once in a while [_] About half of the time [_] All or most of the time 


4. With which of the following groups do you most strongly identify? 


[_] American Indian or Alaska Native [_] Hispanic / Latino/a 


[_] Asian or Pacific Islander [_] White or Caucasian 
[_] Black or African American (1 Other, please describe 
Thank you! 
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Appendix E: 
Web Developer Background Survey 


Please check one of the four boxes for each item below. 


1. 1can always manage to solve difficult problems if I try hard enough. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


2. If someone opposes me, I can find the means and ways to get what I want. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


3. It is easy for me to stick to my aims and accomplish my goals. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


4.1 am confident that I could deal efficiently with unexpected events. 


[_] Not at all true [_] Hardly true [_] Moderately true —[_] Exactly true 


5. Thanks to my resourcefulness, I know how to handle unforeseen situations. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


6. I can solve most problems if I invest the necessary effort. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 


7. can remain calm when facing difficulties because I can rely on my coping abilities. 


[_] Not at all true [_] Hardly true [_] Moderately true — [_] Exactly true 


8. When I am confronted with a problem, I can usually find several solutions. 


[_] Not at all true [_] Hardly true [_] Moderately true —_[_] Exactly true 
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9. If 1am in trouble, I can usually think of a solution. 


[_] Not at all true [_] Hardly true [_] Moderately true 


10. I can usually handle whatever comes my way. 


[_] Not at all true [_] Hardly true [_] Moderately true 


Education 
1. What degrees or certificates do you hold? (Check all that apply) 


[_] High school diploma or GED 
[_] Certificate in: 


[_] Exactly true 


[_] Exactly true 


[_] Two-year or Associate’s degree in: 


[_] Four-year or Bachelor’s degree in: 


[_] Master’s degree in: 


[_] Other, please describe: 


[_] Not applicable 


2. Have you completed any non-degree or self-directed courses in web development? 


If yes, please list course titles or describe in brief. 


3. Do you feel that your education prepared you for your current employment? 


[_] Yes [_] No [_] Mixed 


Please explain why: 
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4. Do you plan to pursue any additional school or training? 


[_] No additional school L_] Vocational or certificate program 
[_] Community college [_] Four-year university 
[_] Other, please describe 


5. Are you currently enrolled in an educational program? (including online, MOOC, degrees, 
etc.) Please list. 


Employment 


1. Which of the following categories best describes your current employment status, in web 
development? 


[_] Employed, working 40 or more hours per [_] Not employed, looking for work 
week 

[_] Employed, working 20-39 hours per week [_] Not employed, NOT looking for work 
[_] Employed, working 1-19 hours per week [_] Other (please describe): 


2. What is your current job title? If not currently employed, what was your most recent job title? 


3. Do you currently have other occupations not related to web development? If yes, please list 


them below and write the number of hours per week for each occupation. If no, please write 
N/A. 


Other Current Occupations: Number of hours per week: 
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4. What are the most important reasons you are interested in the job area you chose? (Please 
check up to three reasons) 


[_] Interest in field 
[_] Location 


[_] Salary/compensation 


[_] Other, please describe 


[_] Less training or additional school needed 
[_] Rapid growth/number of jobs available 
[_] Motivated to help society/people 


5. How many years/months have you worked in web development? 


Years 


Months 


6. Please indicate the overall length of time in years and months that you worked in each role or 
worked with each skill in the table below. Please give an example of a program or skill for 
each area (For example: CSS for Web design, graphics, GUI). If you haven’t had experience 
in arole or skill, please write N/A. 


Role/Skill 


Years/ 
Months 


Example 


Managing a team 


Working individually 


Programming 


Web Design/ graphics/ 
GUI 


Back-end 


7. Please indicate your overall skill level for each role and skills related to web development 
below by using a scale ranging from “1” for novice to “4” for expert. 


Role/Skill 


Novice 
1 


Beginner Intermediate | Expert 
2 3 4 


Managing a team 


Programming 


Web Design/ graphics/ GUI 


Back-end 
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Background 


1. How old are you in years? 


2. Gender: 


3. Growing up, how often did people in your home talk to each other 
in a language other than English? 


[_] Never [_] Once in a while [_] About half of the time [_] All or most of the time 


4, With which of the following groups do you most strongly identify? 


[_] American Indian or Alaska Native [_] Hispanic / Latino/a 


[_] Asian or Pacific Islander [_] White or Caucasian 
[_] Black or African American (1 Other, please describe 
Thank you! 
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Appendix F: 
Survey Results 


Table F1 
General Self-Efficacy (GSE) Scale Means and Standard Deviations by Subgroup 


Community 
High school college Workforce 

Item (n= 102-103) (n=114-115) (n= 83-84) 
| can always manage to solve difficult problems if | try 3.10 (.52) 3.23 (.61) 3.48 (.50) 
hard enough. 
If someone opposes me, | can find the means and 2.97 (.59) 2.88 (.77) 2.79 (.68) 
ways to get what | want. 
It is easy for me to stick to my aims and accomplish 3.17 (.63) 3.31 (.69) 3.29 (.69) 
my goals. 
| am confident that | could deal efficiently with 3.18 (.65) 3.34 (.75) 3.43 (.65) 
unexpected events. 
Thanks to my resourcefulness, | know how to handle 3.12 (.65) 3.37 (.63) 3.46 (.59) 
unforeseen situations. 
| can solve most problems if | invest the necessary 3.39 (.63) 3.59 (.53) 3.67 (.47) 
effort. 
| can remain calm when facing difficulties because | 3.04 (.75) 3.39 (.71) 3.46 (.65) 
can rely on my coping abilities. 
When | am confronted with a problem, | can usually 3.18 (.67) 3.30 (.70) 3.49 (.55) 
find several solutions. 
If | am in trouble, | can usually think of a solution. 3.28 (.62) 3.47 (.65) 3.49 (.53) 
| can usually handle whatever comes my way. 3.22 (.59) 3.40 (.62) 3.45 (.52) 
Total GSE 31.66 (3.41) 33.31 (4.23) 33.99 (3.82) 


Note. There were a few missing cases. Total GSE was calculated only when participant provided a valid response to 
all 10 items. 
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Table F2 


Key Demographic Indicators for High School Profiles, Percentage by School 


Most frequently Parent with 
Female reported bachelor’s degree 
Context n (%) ethnicity (%) or higher (%) 
School A: small charter school in a large city 17 41.2 Hispanic (94.1) 0.0 
School B: large public high school in a small 25 44.0 Hispanic (94.1) 16.0 
city 
School D: large public high school in a small 31 51.6 Two or more 29.0 
city ethnicities (32.3) 
School E: a large public high school in a large 30 70.0 Hispanic (53.3) 32.3 


city 


57 


Table F3 
High School Demographic Frequencies by School 


SchoolA SchoolB SchoolD SchoolE All schools 


Background variables (n=17) (n=25) (n=31) (n=30) (%) 
Age (n = 103) 
16 1 1 1 4 7 (6.8) 
17 16 19 27 25 87 (84.5) 
18 0 5 3 1 9 (8.7) 


Gender (n = 102) 
Female 6 11 16 21 54 (52.4) 
Male 10 14 15 9 A8 (46.6) 


Working computers/laptops at home (n = 103) 


0 0 2 0 2 4 (3.9) 
1 8 3 5 11 28 (26.2) 
2 6 9 ie 9 35 (34.0) 
3 2 7 44 2 22 (21.4) 
4 or more 1 4 4 5 14 (13.6) 
Ethnicity (n =103) 

American Indian or Alaska Native 0 0 0 0 0 

Asian or Pacific Islander 1 2 0 1 4 (3.9) 
Black or African American 0 2 8 7 17 (16.5) 
Hispanic/Latina/o 15 9 9 16 49 (47.6) 
White or Caucasian 1 5 4 1 11 (10.7) 
Other 0 1 0 0 1 (1.0) 
Two or more ethnicities selected 0 6 10 5 21 (20.4) 
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Table F4 
High School Self-Reported Grades Frequencies by School 


SchoolA SchoolB SchoolD SchoolE All schools 
Grades (n=17) (n=25) (n=31) (n=30) (%) 


All classes (n = 101) 


Mostly A’s 3 0 4 6 13 (12.9) 
Mostly A’s and B’s 7 16 12 10 45 (44.6) 
Mostly B’s 3 5 3 3 14 (13.9) 
Mostly B’s and C’s 4 2 9 10 25 (24.8) 
Mostly C’s or lower 0 0 3 1 4 (4.0) 
English classes (n = 102) 
Mostly A’s 4 5 14 12 32 (31.4) 
Mostly A’s and B’s 7 12 9 7 35 (34.3) 
Mostly B’s 5 5 5 4 19 (18.6) 
Mostly B’s and C’s 1 2 3 5 11 (10.8) 
Mostly C’s or lower 0 0 3 2 5 (4.9) 
Math classes (n = 102) 
Mostly A’s 5 5 6 3 19 (18.6) 
Mostly A’s and B’s 5 6 7 9 27 (24.5) 
Mostly B’s 1 2 5 5 13 (12.7) 
Mostly B’s and C’s 4 7 6 9 26 (25.5) 
Mostly C’s or lower 2 5 7 4 18 (17.6) 
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Table F5 
High School Interest in Bright Outlook Careers, Frequencies by School 


SchoolA SchoolB SchoolD SchoolE All schools 
Career (n=17) (n=25) (n=31) (n=30) (%) 


Medical related 


Registered nurse 4 3 4 3 14 (16.5) 
Dental hygienist 0 1 0 1 2 (2.4) 
Biomedical engineer 1 3 1 2 7 (8.2) 
Genetic counselor 0 2 0 0 2 (2.4) 
Physical therapist 2 3 2 2 9 (10.6) 
Medical assistant 3 3 2 5 13 (15.3) 
Emergency medical technician 0 2 3 0 5 (5.9) 
Tech-related 
Web administrator 0 0 1 0 1 (1.2) 
Web developer 2 2 3 0 7 (8.2) 
Computer and information systems 2 1 2 0 5 (5.9) 
manager 
Video game designer 2 3 5 0 10 (11.8) 
Computer systems analyst 3 2 0 0 5 (5.9) 
Unrelated 
Tutor 1 0 2 0 3 (3.5) 
Music therapist 0 0 0 0 1 (1.2) 
Spa manager 1 0 0 0 1 (1.2) 
Climate change analyst 0 0 0 0 0 (0.0) 


Note. Students selected more than one. 


Survey results pertaining to the web development community college participants are 
also presented by bootcamp participants (Site J) and traditional community college participants 
(Sites F and H). 
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Table F6 


Frequency and Percentage of Community College Response to Survey Question, “Are 
your courses conducted online or in person?” by Career Domain 


Web development 


(n = 47) 
EMT Bootcamp Traditional 
Response (n = 62) (n = 19) (n = 28) Total 
Online only 0 0 1 (100.0) 1 (100.0) 
In person only 38 (65.51) 7 (12.06) 13 (22.41) 58 (100.0) 
Online and in person 25 (49.01) 12 (23.52) 14 (27.45) 51 (100.0) 


Note. Two EMTs did not report courses. 


Table F7 


Community College Responses to Survey Question, “Please list the courses you have completed relevant 
to your specific program of study,” by Career Domain 


Web development 


EMT Bootcamp Traditional 
Response (n = 63) (n = 19) (n = 28) 
Mean number of courses reported 1.82 1.31 2.92 
Examples Anatomy, CPR, Fire Web fundamentals, Computer Science, 
Tech, Physiology Python HTML, CSS, Java 


Note. One EMT did not report courses. 
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Table F8 


Frequency and Percentage of Community College Future Schooling Plans by Career Domain 


Web development 


(n = 47) 
EMT? Bootcamp? Traditional® Total 
(n = 63) (n = 19) (n = 28) (n = 109) 
No additional schooling 12 (46.0) 7 (26.9) 7 (26.2) 26 (100.0) 
Community college 9 (81.8) 0 2 (18.2) 11 (100.0) 
Vocational or certificate program 9 (56.3) 2 (12.5) 5 (31.3) 16 (100.0) 
Four-year university 18 (56.3) 2 (6.3) 12 (37.5) 32 (100.0) 
Other 15 (42.9) 8 (32.0) 2 (8.0) 25 (100.0) 


Note. Only the highest degree or the “other” category are reported. 

*For Other, EMT students reported paramedic school, Master’s in physician assistant, medical school, or grad 
school, or “possibly, undecided.” Vocational programs were Fire Academy or paramedic school. One EMT student 
did not answer this question. 

>For Other, bootcamp web development students reported languages, online courses or MOOCs, MBA, or Master’s 
degree/PhD in data science, or “possibly.” Vocational programs were unspecified. 

‘For Other, community college web development students reported MOOC or unspecified grad school. Vocational 
programs were unspecified. 


Table F9 


Community College Highest Education Attained by Career Domain and School 


Web development 


(n = 47) 
EMT Bootcamp Traditional Total 
(n = 64) (n= 19) (n = 28) (n= 111) 
High school or none 38 3 15 56 
Certificate 4 0 4 8 
Associate’s 2 5 1 8 
Bachelor’s 18 9 7 34 
Master’s 1 1 1 3 
Other 1 1 0 2 
All 64 19 28 111 


Note. It should be observed that in both career domains, participants frequently reported 
multiple lower degrees or certifications. Only the highest degree or the “other” category 
are reported. 
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Table F10 


Frequency and Percentage of Community College Current Employment Status by Career Domain 


Web development 


(n = 46) 
EMT Bootcamp _ Traditional Total 
Employment status (n = 63) (n = 19) (n = 28) (n = 110) 
Employed, working 40 or more hours a week 5 (38.5) 0 8 (61.5) 13 (100.0) 
Employed, working 20-39 hours a week 17 (77.3) 0 5 (22.7) 22 (100.0) 
Employed, working 1-19 hours a week 17 (63.0) 2 (7.7) 7 (26.9) 26 (100.0) 
Not employed, looking for work 14 (56.0) 5 (20.0) 6 (24.0) 25 (100.0) 
Not employed, NOT looking for work 8 (38.1) 11 (52.4) 2 (9.5) 21 (100.0) 
Other 2 (8.7) 1 (4.3) 0 23 (100.0) 


Note. One EMT student did not answer this question. Employment refers to jobs not related to their current field 
of study. 


Table F11 


Frequency and Percentage of Workforce Highest Education 
Attained by Career Domain 


Highest education EMT Web developer 
attained (n = 45) (n = 39) 
High school 7 (15.6) 5 (12.8) 
Certificate 17 (37.8) 1 (25.6) 
Associate 11 (24.4) 0 (0.0) 
Bachelor’s 10 (22.2) 25 (64.1) 
Master’s 0 (0.0) 6 (15.4) 
Other 0 (0.0) 0 (0.0) 
All 45 (100.0) 39 (100.0) 


Note. It should be observed that in both career domains, 
participants frequently had multiple lower degrees or 
certifications. For example, 30 EMTs reported certificates but a 
degree “outranked” a certificate. For the web development group, 
participants listed many previous and current enrollments in online 
skills classes, as well as some programs and certification courses. 
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Table F12 


Frequency and Percentage of Workforce Response to Survey Question, 
“Do you feel that your education prepared you for your current 
employment?” by Career Domain 


EMT Web developer Total 
Response (n=45) (n = 39) (n = 84) 
Yes 30 (66.7) 13 (33.3) 43 (51.2) 
No 7 (15.6) 3 (7.7) 10 (11.9) 
Mixed 8 (17.8) 23 (59.0) 31 (36.9) 


Table F12 shows the frequency and percentage of participant responses to the question, 
“Do you feel that your education prepared you for your current employment?” Participants 
were also asked to explain why. Below are a few examples of participant responses: 
EMT: Yes. It prepared me for about 60% of my jobs as an EMT. However, 
as medicine is an art (something that will always be evolving), | think my 


education has given me resources to critical think and problem solve for 
what | am not prepared for. 


EMT: No. Ed[ucation] system focuses on passing tests and not thorough 
understanding. 


EMT: Mixed. | feel most pertinent skills are learned on the job, while 
education provides an intro/opens doors to those jobs. 


Web Developer: Yes. Education helped me develop problem-solving skills 
and programming skills. 


Web Developer: No. | believe my core aptitudes in logic reasoning 
prepared me. | am working outside my field of study. 


Web Developer: Mixed. Textbook and online tutorials don’t accurate 
meet real world scenarios. Learned by working. 
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Table F13 


Frequency and Percentage of Workforce 
Response to Survey Question, “When was 
the last time you completed continuing 
education courses to renew your EMT 
license?” (n = 44) 


Year Frequency % 
Never 3 6.7 
2018 23 51.1 
2017 13 28.9 
2016 5 11.1 


Note. There was one missing case. 


Table F14 


Frequency and Percentage of Workforce Response to Survey Question, 
“Are you currently enrolled in an educational program?” by Career 
Domain 


EMT Web developer Total 
Response (n=45) (n = 39) (n = 84) 
Yes 34 (75.6) 12 (30.8) A7 (58.3) 
No 11 (24.4) 27 (69.2) 38 (45.2) 


Note. For EMTs, this question specified “other than continuing education 
courses.” For web developers, this question gave “online, MOOC [massive 
online open course], degrees, etc.” as examples. 


Table F15 


Frequency and Percentage of Currently 
Enrolled Educational Programs Reported 
by Workforce EMTs (n = 45) 


Program Frequency % 
Paramedic 25 55.6 
Other 9 20.0 
None 11 24.4 


Note. Other educational programs included 
associate’s or bachelor’s degrees in various 
fields, including biology, business 
administration, health science, Middle 
Eastern studies, linguistics, neuroscience, 
and public safety. 
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Table F16 


Frequency and Percentage of Workforce Future Schooling Plans by Career Domain 


EMT Web developer Total 
Future schooling plans (n= 45) (n = 39) (n = 84) 
No additional schooling 1 (2.2) 19 (48.7) 20 (23.8) 
Community college 3 (6.7) 0 (0.0) 3 (3.6) 
Vocational or certificate program 6 (13.3) 6 (15.4) 12 (14.3) 
Four-year university 19 (42.2) 0 (0.0) 19 (22.6) 
Other 16 (35.6)? 14 (35.9)° 30 (35.7) 


Note. Only the highest degree or “other” are reported. 

*For Other, EMTs reported paramedic school, master’s in computational linguistics, master’s in 
physician assistant, Ph.D. in biochemistry, medical school, or grad school (unspecified field), or 
“possibly.” 

>For Other, web developers reported online courses, project management, master’s degree 
(unspecified field), code academy, Amazon Web Services certification, or “don’t know yet” or “if the 
opportunity arises.” 


Table F17 
Frequency and Percentage of Current Employment Status as an EMT/Web Developer by Career Domain 


EMT Web developer Total 
Employment status (n = 45) (n = 39) (n = 84) 
Employed, working 40 or more hours a week 14 (31.1) 29 (74.4) 43 (51.2) 
Employed, working 20-39 hours a week 9 (20.0) 0 (0.0) 9 (10.7) 
Employed, working 1-19 hours a week 7 (15.6) 6 (15.4) 13 (15.5) 
Not employed, looking for work 1 (2.2) 4 (10.3) 5 (6.0) 
Not employed, NOT looking for work 14 (31.1) 0 (0.0) 14 (16.7) 
Other 0 (0.0) 0 (0.0) 0 (0.0) 


Note. Participant responses may not be accurate. For instance, 24 of the EMTs were recruited from a full-time 
paramedic program in which, during informal conversation, we learned that they had all recently resigned to 
enroll; however, responses from the 24 varied widely. Additionally, three EMT respondents wrote in “recently 
resigned” or “full time school” or equivalent and were recoded into the “Not employed, not looking for work” 
category. 
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Table F18 


Frequency and Percentage of Workforce Response to Survey Question, “Do 
you currently have other occupations [aside from EMT/not related to web 
development] ?” by Career Domain 


EMT Web developer Total 
Response (n=45) (n = 39) (n = 84) 
Yes 15 (33.3) 5 (12.8) 20 (23.8) 
No 26 (57.8) 20 (51.3) A6 (54.8) 
Not reported 4 (8.9) 14 (35.9) 18 (21.4) 


Note. For EMTs, other occupations included clinical or skills instructor, self- 
employed, tutor, lifeguard, ride-share driver, firefighter, handyman/plumber, 
dental technician, dishwasher, resident adviser, and performing artist. For web 
developers, other occupations included ride-share driver, electric scooter charger, 
film/TV composer, producer/artist relations, andenterprise resource planning 
developer. 


Table F19 


Frequency and Percentage of Workforce Response to Survey Question, “What are 
the most important reasons you are interested in the job area you chose?” by Career 
Domain 


EMT Web developer 
Response (n=45) (n = 39) 
Interest in field 40 (88.9) 30 (76.9) 
Location 6 (13.3) 7 (17.9) 
Salary/compensation 15 (33.3) 25 (64.1) 
Less training or additional school needed 0 (0.0) 5 (12.8) 
Rapid growth/number of jobs available 8 (17.8) 19 (48.7) 
Motivated to help society/people 40 (88.9) 6 (15.4) 
Other 15 (33.3)? 1 (2.6)> 


Note. Responses are not mutually exclusive. Participants were asked to choose up to three 
reasons, however a few marked four or five. 

*For Other, EMT responses included passion, lifestyle, fits personality, stepping stone for 
future jobs, fire, experience in emergency settings, experience applicable to medicine, 
personal reasons, excitement, and brotherhood. 

>For Other, the web developer response was “I like puzzles.” 
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Table F20 


Workforce Descriptive Results of Total Months Working in an Occupation by 
Career Domain 


Response n Minimum Maximum M SD 
EMT 45 9 140 34.0 25.3 
Web developer 39 6 270 87.2 77.2 
Table F21 


Frequency and Percentage of Workforce Experience 
Managing a Team in Occupation by Career Domain 


EMT Web developer 
Response (n = 45) (n = 39) 
Yes 9 (20.0) 20 (51.3) 
No 36 (80.0) 19 (48.7) 


Note. Responses to other skills for web developers are not 
reported due to inconsistencies in participant responses. 


Table F22 


Frequency and Percentage of Workforce Response to Survey Question, “Growing 
up, how often did people in your home talk to each other in a language other 
than English?” by Career Domain 


EMT Web developer Total 
Response (n = 45) (n = 39) (n = 84) 
Never 17 (37.8) 9 (23.1) 26 (31.0) 
Once in awhile 6 (13.3) 7 (17.9) 13 (15.5) 
About half of the time 8 (17.8) 13 (13.3) 21 (25.0) 
All or most of the time 13 (28.9) 10 (25.6) 23 (27.4) 
Not reported 1 (2.2) 0 (0.0) 1 (1.2) 
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